{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Semantic Layer on CSV\n",
    "\n",
    "In this notebook, we will show how to create a semantic layer on a CSV file.\n",
    "The semantic layer works as a bridge between the raw data and the natural language layer.\n",
    "\n",
    "### Why use a Semantic Layer?\n",
    "- Adds context and meaning to data columns\n",
    "- Makes it easier for the large language model to understand context\n",
    "- Set once, use across multiple sessions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import PandasAI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandasai as pai"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read raw data\n",
    "\n",
    "For this example, we will use a small dataset of heart disease patients from [Kaggle](https://www.kaggle.com/datasets/arezaei81/heartcsv)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the heart disease dataset\n",
    "file_df = pai.read_csv(\"./dataheart.csv\")\n",
    "\n",
    "# Display the first few rows\n",
    "file_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create the Semantic Layer\n",
    "\n",
    "Requirements for the semantic layer:\n",
    "- `path`: Must be in format 'organization/dataset'\n",
    "- `name`: A descriptive name for the dataset\n",
    "-  `df`: A dataframe\n",
    "- `description`: Brief overview of the dataset\n",
    "- `columns`: List of dictionaries with format:\n",
    "  ```python\n",
    "  {\n",
    "      \"name\": \"column_name\",\n",
    "      \"type\": \"column_type\",  # string, number, date, datetime\n",
    "      \"description\": \"column_description\"\n",
    "  }\n",
    "  ```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = pai.create(path=\"organization/heart\",\n",
    "    name=\"Heart\",\n",
    "    description=\"Heart Disease Dataset\",\n",
    "    df = file_df,\n",
    "    columns=[\n",
    "        {\n",
    "            \"name\": \"Age\",\n",
    "            \"type\": \"integer\",\n",
    "            \"description\": \"Age of the patient in years\"\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"Sex\",\n",
    "            \"type\": \"string\",\n",
    "            \"description\": \"Gender of the patient (M: Male, F: Female)\"\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"ChestPainType\",\n",
    "            \"type\": \"string\",\n",
    "            \"description\": \"Type of chest pain (ATA: Atypical Angina, NAP: Non-Anginal Pain, ASY: Asymptomatic, TA: Typical Angina)\"\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"RestingBP\",\n",
    "            \"type\": \"integer\",\n",
    "            \"description\": \"Resting blood pressure in mm Hg\"\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"Cholesterol\",\n",
    "            \"type\": \"integer\",\n",
    "            \"description\": \"Serum cholesterol in mg/dl\"\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"FastingBS\",\n",
    "            \"type\": \"integer\",\n",
    "            \"description\": \"Fasting blood sugar (1: if FastingBS > 120 mg/dl, 0: otherwise)\"\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"RestingECG\",\n",
    "            \"type\": \"string\",\n",
    "            \"description\": \"Resting electrocardiogram results (Normal, ST: having ST-T wave abnormality, LVH: showing probable or definite left ventricular hypertrophy)\"\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"MaxHR\",\n",
    "            \"type\": \"integer\",\n",
    "            \"description\": \"Maximum heart rate achieved\"\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"ExerciseAngina\",\n",
    "            \"type\": \"string\",\n",
    "            \"description\": \"Exercise-induced angina (Y: Yes, N: No)\"\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"Oldpeak\",\n",
    "            \"type\": \"float\",\n",
    "            \"description\": \"ST depression induced by exercise relative to rest\"\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"ST_Slope\",\n",
    "            \"type\": \"string\",\n",
    "            \"description\": \"Slope of the peak exercise ST segment (Up, Flat, Down)\"\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"HeartDisease\",\n",
    "            \"type\": \"integer\",\n",
    "            \"description\": \"Target variable - Heart disease presence (1: heart disease, 0: normal)\"\n",
    "        }\n",
    "    ])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Semantic Dataframe\n",
    "\n",
    "Once you have saved the dataframe with its semantic layer, you can load it in any session using the `load()` method. This allows you to:\n",
    "- Maintain data context across sessions\n",
    "- Ask questions about your data in natural language\n",
    "- Generate more accurate analysis and visualizations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the semantically enhanced dataset\n",
    "df = pai.load(\"organization/heart\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Chat with your dataframe\n",
    "\n",
    "You can now ask questions about your data in natural language to your dataframe using the `chat()` method. PandasAI can be used with several LLMs. For the purpose of this example, we are using LiteLLM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pandasai_litellm.litellm import LiteLLM\n",
    "\n",
    "# Initialize LiteLLM with your OpenAI model\n",
    "llm = LiteLLM(model=\"gpt-4.1-mini\", api_key=\"YOUR_OPENAI_API_KEY\")\n",
    "\n",
    "# Configure PandasAI to use this LLM\n",
    "pai.config.set({\n",
    "    \"llm\": llm\n",
    "})\n",
    "\n",
    "response = df.chat(\"What is the correlation between age and cholesterol?\")\n",
    "\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
